Dynamic variation of output media signal in response to input media signal

ABSTRACT

A first input signal, which has a first playback sequence, is resequenced based upon a characteristic of a second input signal. The resequencing occurs analyzing the characteristic in the second input signal, and modifying the first playback sequence of the first input signal based upon the analysis of the characteristic to generate a second playback sequence. Finally, a third signal is output using the second playback sequence.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. provisional patentapplication No. 60/322,944 entitled “System and Method for CreatingMultimedia”, filed Sep. 15, 2001, in the names of NEUMAN et al., thedisclosure of which is expressly incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to multimedia. More specifically,the present invention relates to creating an output media signal basedupon an input media signal.

[0004] 2. Background Information

[0005] The combination of audio and video has become a popular means ofentertainment in film, television, video games, computers, PDAs, theinternet, kiosks, karaoke, ambient environments, and live events. It isextremely desirable to correlate a moving image to a sound track, orvice versa. This technique is demonstrated in film, television, musicvideos, video games, web sites, cartoons, clubs, events, concerts, lightshows, CD-ROMs, art installations, etc.

[0006] Current technology requires the user to manually edit a movingpicture with a soundtrack, either by editing frames or altering theplayback rate of the audio or video. This process requires highlyskilled users to work with complex editing tools and understand musicstructure so as to edit in synchronicity with a sound track. Thistraditional editing process is both very expensive and time consuming.In fact, one of the more costly and time consuming aspects ofprofessional production of film, video, and multimedia is synchronizingaudio and video sources.

[0007] Recent developments in home consumer level systems capable ofplaying and manipulating video has created a demand for an editingsystem that creates motion pictures synchronized to an audio track. Thisprocess is too complicated for the average user to perform utilizingcurrent technologies. Under the status quo, a user has to correlate anaudio track to moving images by either:

[0008] 1. Simultaneously capturing audio and motion pictures, which hassignificant problems; In order to have a correlation between the imagesand the audio, elaborate pre-production choreography is required;Another disadvantage of this simultaneous audio/video capture is thatthe audio and video become dependent; This restricts the user fromeasily changing either the audio or video of the synchronized piece; or

[0009] 2. Manually editing a motion picture, cutting scenes in relationto changes in a soundtrack; This technique requires the user to detectappropriate cues in a soundtrack, such as a snare drum, and then editthe picture at that moment in time; Changes in the picture or audioportions of the program require manually re-setting these in and outpoints, or utilizing a semi-automated technology whereby the user has tomanually correlate the sound cue to the desired frame and lock therelationship in the time line.

[0010] At live events, video is frequently used to enhance the viewer'sexperience. Current technology allows only basic automated visualeffects in response to audio. This is demonstrated by strobing lightsand lasers triggered by audio characteristics or, in the case of video,image manipulation of color, hues or similar attributes. More complexcorrelations between audio and motion pictures typically utilize aprocess in which a user pre-programs video effects, and manuallytriggers such effects through an input device, such as a keyboard,mouse, or trackball. The status quo lacks tools for generating real timevideo content derived from dynamic media signals, which do not requireanalysis or input from a user.

[0011] The creation of a series of still images synchronized with asoundtrack requires manually setting the image frame rate to the tempoof the audio. Changing the audio to synchronize to the picture requiresthe user to edit the sound or manually alter the playback rate. Audiowith an inconsistent tempo requires the user to manually determine wheresignificant audio cues reside and edit the still images to thissequence. Any changes in audio require the process to be repeated forthe modified portion.

[0012] The ability to quickly, simply and iteratively produce new mediacontent is of special interest in contexts such as movie making. Inparticular, home consumer production of movie content suffers from thelack of easy to use yet powerful composition tools that do not requirevideo or audio analysis by the user. Currently there are no tools thatallow the user to easily synchronize or edit a motion picture or seriesof still images to an unrelated audio track or tracks. In addition,there is no mechanism by which pre-existing recordings can be easily andefficiently combined to present the desired effect. This is particularlytrue for non-professionals.

[0013] In summary, there is a need for a time-based media processingsystem that is capable of providing high-quality, adaptive mediaproductions without requiring a significant level of skill on the partof the user, and is therefore suited for use by both the professionaland average consumer. There would be great utility in a tool in whichtwo unrelated media signals can be combined to create a motion picturesynchronized to a sound track or other media signals and vice versa.

SUMMARY OF THE INVENTION

[0014] According to an aspect of the invention a method and a computerreadable medium are provided for resequencing a first input signal,which has a first playback sequence, based upon at least onecharacteristic of a second input signal. The method and medium bothinclude analysis of the characteristic of the second input signal, andmodifying of the first playback sequence of the first input signal togenerate a second playback sequence. The modification occurs based uponthe analysis. In addition, a third signal including the second playbacksequence is output.

[0015] In one embodiment, the first input signal is a video signal, thesecond input signal is an audio signal, and the third signal is a videosignal. In another embodiment, the first input signal and the secondinput signal are the same signal. It is possible that the analyzing andthe outputting occur in real time. Further, the second input signal canbe output along with the third signal. The analyzing can occur at afrequency based upon a frame rate of the first input signal. Themodifying can also include altering a frame rate.

[0016] In another embodiment, the first input signal is actuallymultiple input video signals, each having a linear playback sequence,and the third signal is actually multiple output video signals. Thecharacteristic is actually multiple characteristics, each beingassociated with one of the input video signals. In this embodiment, themodifying requires modifying of a linear playback sequence of each ofthe input video signals based upon the analysis of the associatedcharacteristics to generate an output sequence. The outputting alsorequires outputting each output video signal in accordance with itsoutput sequence.

[0017] In another embodiment, the first input signal includes multiplelayers, as does the third signal. Similarly, the characteristic includesmultiple characteristics, each being associated with one of the layers.In this embodiment, the modifying includes modifying a layer playbacksequence of each of the layers based upon the analysis of the associatedcharacteristics to generate an output layer sequence. Finally, theoutputting requires outputting of each layer in accordance with itsoutput layer sequence.

[0018] According to another aspect, a method and a computer readablemedium are provided for resequencing a first input signal based upon atleast one characteristic of a second input signal. The first inputsignal has a series of frames, each having a frame number. The methodand computer readable medium include analyzing the characteristic of thesecond input signal. They also include calculating, for each frame inthe series of frames, an offset based upon the analyzing. The offset isthen added to the frame number to obtain an output frame number.Finally, a third signal is output in a playback sequence using theoutput frame numbers.

[0019] According to another aspect, a method and a computer readablemedium are provided for outputting a signal based upon at least onecharacteristic of an input signal. The method and medium requireanalysis of the at least one characteristic of the input signal. Theyalso require control of an external device based upon the analysis ofthe at least one characteristic, and output of the signal using thecontrolled external device.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention is further described in the detaileddescription that follows, by reference to the noted drawings by way ofnon-limiting examples of embodiments of the present invention, in whichlike reference numerals represent similar parts throughout several viewsof the drawing, and in which:

[0021]FIGS. 1A, 1B, and 1C are graphs plotting source video, sourceaudio and output video, respectively, against time;

[0022]FIG. 2 shows exemplary video frame substitution based upon anaudio signal, according to an aspect of the present invention; and

[0023]FIG. 3 is a flow chart showing exemplary processing to determinean output stream, according to an aspect of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0024] In view of the foregoing, the present invention, through one ormore of its various aspects, embodiments and/or specific features orsub-components, is thus intended to bring out one or more of theadvantages as specifically noted below.

[0025] In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be evident, however, toone skilled in the art that the present invention may be practicedwithout the specific details. In other instances, well-known structuresand devices are shown in block diagram form to facilitate explanation.The description of preferred embodiments is not intended to limit thescope of the appended claims.

[0026] An object of the invention is to enable new efficiencies,methods, and forms in the production of media content. The inventionalso aims to satisfy a need for a media processing system that reducesthe labor and expertise necessary to create media content. Real timeapplications of this invention can immediately provide visual feedbackfrom audio input as an essential aid to the production and creation ofmultimedia.

[0027] The invention may be used to help visualize audio segments withnon associated video segments. Individual tracks can be visualized tohelp illustrate the behavior of a musical instrument, a voice or a soundeffect. Multiple images can be shown simultaneously to separate, similarto a graphic equalizer, the complexity and dynamics of an audio segment.In another embodiment, video images may be layered with each layercontaining a separate object. Each layer may be individually controlledby the system so that different objects are graphically manipulated bydifferent discrete media signals.

[0028] Aspects of the present invention may be implemented on one ormore computers executing software instructions, or may be encoded ontoread only memory, PROM, EPROM, firmware, or circuitry for use in anelectronic apparatus. According to one embodiment of the presentinvention, server and client computer systems transmit and receive dataover a computer network or standard telephone line. The steps ofaccessing, downloading, and manipulating the data, as well as otheraspects of the present invention are implemented by central processingunits (CPUs) in the server and client computers executing sequences ofinstructions stored in a memory. The memory may be a random accessmemory (RAM), read only memory (ROM), a persistent storage, such as amass storage device, or any combination of these devices. Execution ofthe sequences of instructions causes the CPU to perform steps accordingto embodiments of the present invention.

[0029] The instructions may be loaded into the memory of the server orclient computers from a storage device or from one or more othercomputer systems over a network connection. For example, a clientcomputer may transmit a sequence of instructions to the server computerin response to a message transmitted to the client over a network by theserver. As the server receives the instructions over the networkconnection, it stores the instructions in memory. The server may storethe instructions for later execution, or it may execute the instructionsas they arrive over the network connection. In some cases, thedownloaded instructions may be directly supported by the CPU. In othercases, the instructions may not be directly executable by the CPU, andmay instead be executed by an interpreter that interprets theinstructions. In other embodiments, hardwired circuitry may be used inplace of, or in combination with, software instructions to implement thepresent invention. Thus, the present invention is not limited to anyspecific combination of hardware circuitry and software, nor to anyparticular source for the instructions executed by the server or clientcomputers.

[0030] In one embodiment, the invention alters a single media signal ora combination of two or more media components into a modifiedpresentation. In an embodiment, the system analyzes a second media fileand samples a characteristic of the second media file. For example, thesecond media file may be an audio file that is analyzed for any soundsthat exceed a pre-set decibel threshold. In this example, a derivativefile would store all times when the decibel threshold has been exceeded,as well as the amount by which the decibel levels were exceeded. If thesecond media file is a stream, i.e., real time, instead of a file, thederivative file would simply be a single value resulting from a realtime analysis occurring at the sampled frequency.

[0031] An output media file is then formed based upon the first mediasignal and the analysis of the second media signal (i.e., the derivativefile). That is, the derivative file is used to modify the output file.Additionally, the derivative file can be used to control an externaldevice containing the first media signal. Examples of this device are adirect disk recorder (DDR), such as TIVO, and a DVD player. Finally, thesecond media file and the output media file are played together.Specifically, when the derivative file so indicates, the systemrearranges the playback sequence of frames of the first media file. Inone embodiment, the output signal is a modified media presentationhaving multiple media components.

[0032] An example is now provided with reference to FIGS. 1A, 1B, and 1Cin which a sequence of frames in a video signal is altered based upon adetected amplitude of an audio signal. FIG. 1A shows standard sourcevideo playback as a linear sequence of frames over time. FIG. 1B showsan amplitude of an audio signal as it varies with time. As shown in FIG.1C, the output signal is created from a non-linear reassembly of framesof the source video so that each frame correlates the amplitude of theaudio with the future/past frames of the source video.

[0033] In an audio and video media signal embodiment, the video signalmay include moving objects appearing to move in a non-linear manner insync with a media, which may be the audio file. For example the movementof the objects in the video may appear to move to the beat of the audiotrack.

[0034] Normally, audio and video files are played in a specific sequencethat replicates a recorded source subject, which may be a musicalinstrument performance or a movement of objects. The sequence of soundsreplicates music and the sequence of pictures shows a visualrepresentation of moving figures. Electronic audio files have a presetsampling rate at which specific sounds are sequentially emitted by anaudio file player. Similarly, video files also have a preset frame rateat which a specific sequence of video frames are displayed by a mediaplayer. Because the audio sample rate and video frame rate may be fairlyhigh, e.g., 160 kbs/second or higher and 30 frames/second, respectively,the human ears and eyes cannot distinguish the breaks between the tonesand picture frames.

[0035] As an example, the media signal may be a drum sound that definesthe beat of the song of an audio file. The inventive system can definethe drum sound as the characteristic and sample this characteristicwhile playing the audio file normally. When the audio and video tracksare played together, the inventive system will automatically alter thesequence of the video file based upon the characteristic of the audiosignal. Specifically, the video frame that would normally be displayedat the time of the characteristic (i.e., the beat) is not displayed andan alternate substitute video frame is displayed in its place. Inaddition, each and every frame could be reordered.

[0036] In other embodiments, any other type of media signalcharacteristic may be used to trigger the non-linear playback of a mediasignal. For example, because a video signal may be completelyindependent of an audio signal, the picture and sound files used by thesystem can be associated or completely disassociated. An example ofassociated audio and video signals is a musical performance featuringaudio and video recordings of the musical performance. An example ofdisassociated audio and video files is a song and unrelated videofootage of a sporting event. The system does not rely upon synchronizingmarkers in both the audio and video and is therefore compatible with anymedia signal, such as an audio file and a video file, as long as one ofthe files has a characteristic off which the system can trigger.

[0037] In an embodiment, a substitute frame is a frame that is close insequence to the frame that would normally be displayed. For example, thesystem may display a video frame that is a few frames ahead of or behindthe frame that would normally be displayed. This substitution of an outof sequence frame creates an effect where objects appear to be movingnormally in a linear manner and then when the audio signalcharacteristic is detected, a non-sequential video frame or frames aredisplayed between the sequential frames. The non-sequential frames maybe ahead of or behind the normally next sequential frame. If thenon-sequential frame is ahead of the normal sequence, the video willappear to jump forward at the moment of the audio signal characteristic.Similarly, if the non-sequential frame is behind the normal sequenceframe, the video will appear to jump backwards.

[0038] In yet another embodiment, the frame or frames displayed when theaudio signal characteristic is detected may be displayed multiple timescreating a temporary freezing of motion. When the audio characteristichas passed, the video returns to its normal linear playback until thenext discrete audio signal is detected. The effect is a pulsation in thenormally linear motion of the video that is synchronized with the audiosignal characteristic.

[0039] An embodiment will now be described with reference to FIG. 2. Afirst signal 1, which is a video signal in this example, includesmultiple frames. In FIG. 2, frames 11-18 are shown with correspondingframe numbers 11-18 (Frame 10 is only partially shown). A second signal2 is an audio signal in this example. A characteristic of the audiosignal 2, e.g., an amplitude, may be sampled. In FIG. 2, the first threesampling periods 20, 22, 24 the audio signal has an amplitude of 0.Thus, because no characteristic is detected, an output signal 3 is thesame as the input signal 1. In other words, frame numbers 10, 11, and 12are displayed at frames 40, 42, 44. At the time of the fourth sampling26, an amplitude of 0.4 is detected. Thus, a substitute frame 46 (framenumber 17) is displayed instead of the normally displayed frame 13 ofthe video signal 1. In other words, the output signal 3 displays framenumbers 10, 11, 12 and 17, in this order. Display of frame 17 is aheadof the normal sequence and thus, it will appear that the character hasjumped ahead of where he would be expected to be.

[0040] At the fifth and sixth sampling periods 28, 30 the amplitude hasreturned to 0. Thus, the next output frames 48, 50 will again correspondto the video signal 1 at those times (i.e., frame numbers 14 and 15 aredisplayed). At the time of the seventh sampling period 32, an amplitudeof 0.2 is detected. Thus, a substitute frame 52 is displayed instead ofthe normally displayed frame number 16 of the video signal 1. In otherwords, the output signal 3 displays frame numbers 10, 11, 12, 17, 14,15, and 18 in this order. Display of frame 18 is not in the normalsequence and thus, it will appear that the character has again jumpedahead of where he would be expected to be. At the eighth and ninthsampling periods 34, 36 the amplitude has returned to 0. Thus, the nextoutput frames 54, 56 will again correspond to the video signal 1 atthose times.

[0041] Various effect modifications can be made to the described system.For example, the magnitude of the jump in the non-sequential video framecan be controlled by the user of the system. This offset can be dynamicand can be controlled to grow, shrink, oscillate, decay, delay, driftand sustain.

[0042] In an alternative embodiment, a single media signal may be used.In this embodiment, the system detects characteristics in the singlemedia signal. The system may alter the output signal of the single mediasignal based upon the detected characteristics in the same media. Forexample in a video signal, any of the video effects may be applied tothe video media upon the detection of a characteristic in the same videosignal. Thus, the system may utilize a single media source to produce analtered output media signal.

[0043] In another embodiment, a video frame from a video source otherthan the second video signal may be the substituted frame(s) displayedwhen the audio characteristic is detected. In this embodiment, thesecond video file may be any other type of visual data including but notlimited to: a 2-D file, 3-D data, and a live video feed. When the othervideo source is an animation file, the animation file may be controlledby the system so that the graphical display is altered when an audiocharacteristic is detected.

[0044] Because 2-D and 3-D computer graphics programs continuouslydisplay a composite image rather than a series of sequential frames, theinventive system will operate in a different manner when applied to sucha program. For example, the motion of each displayed object may beindividually controlled. Thus, the control of individual components maycorrespond to different characteristics. In an embodiment, a firstobject may be altered by a first characteristic and a second object maybe altered by a second characteristic. For example, the first object mayappear to bounce in response to the first characteristic and the secondobject may be enlarged in response to the second characteristic. Anynumber of visual effects may be applied to the 2-D or 3-D objectsincluding movement, appearance, camera angle, shape, etc.

[0045] In another embodiment, the visual display may be a layeredcomposite of visual frames. Each of the frame layers may contain aseparate object and each layer may be individually manipulated inresponse to detecting of a characteristic. By varying only some of theframes of the composite, only objects illustrated in the layers arevisually altered. For example, the first layer may include a backgroundof mountains, a second layer may include trees and a lake, a third layermay include an airplane and the fourth layer may include a car. Themountain layer may be altered by a bass audio signal, the trees and lakelayer may be altered by a mid frequency audio signal. The airplane layermaybe altered by a high frequency audio signal and the car layer may bealtered by a discrete audio signature. The visual effect would showindividual objects appearing to be independently altered by differentcharacteristics.

[0046] In yet another embodiment, the frame rate can be altered, so thatthe display rate is higher or lower than the normal 30 frames/second. Anoffset calculation determined from signal 2 (described below) can beused to vary the playback rate. When a characteristic is detected, thesystem may temporarily accelerate the playback speed to 150frames/second. The accelerated frames may be the normal sequential videoframes, substitute frames or a visual file from another source ormultiple other sources. The visual effect of displaying multiple framesat a faster rate would be acceleration, because the velocity of objectsdisplayed increases. Conversely, if the frame rate is slowed, the visualeffect would be a deceleration of moving objects. Of course, this framerate effect can be applied to a single region visual display, a layeredvisual display or a multiple region visual display.

[0047] In another embodiment, multiple frames can be displayed in thetime allotted for a single frame. Similarly, partial frames or blendedframes may be inserted as substitute frames in a sequence of videoframes.

[0048] One preferred embodiment of the invention is implemented as a setof software instructions on a computer. In this manifestation, theinvention analyzes an audio signal. that may be either analog ordigital. Exemplary audio formats include stereo, mono, and multipletrack signals, such as MP3, Quicktime, soundedit, MIDI, IMA, WAV, AIFF,Sound designer II, sun .au, or any format that may be developed in thefuture. Sound can be live, pre-recorded, or transmitted through airwavesor dedicated lines, such as the internet, a LAN or a WAN. The audiosignal can contain multiple tracks, and the invention can discernindividual (split) tracks (e.g., —a drum track residing in a multi-trackcomposition or score). The invention can accept signals frommicrophones, audio tapes (e.g., DAT and cassette), video tapes withsound tracks, CD, DVD, MIDI devices, mini disc or any future soundtechnologies.

[0049] The characteristic being detected can also be located in a signalcreated by a user. The user created signal could be employed when theactual audio signal does not provide the intended effect. That is, theuser created signal can be used to achieve the intended effect. In thiscase, the user can select or alter specific characteristics of an audiotrack, for example its tempo, tone, beat, pitch, duration, carrierfrequency, scale or volume. The user can also adjust the amplitudes ofspecific sound frequencies and can shift the pitch of the audio. Inaddition, the user can generate a tone (by specifying a frequency,amplitude and shape) and can insert the tone at any point in time. Sucha user created signal can be recorded or can be live. An alternativeuser-created signal can include a duplicate of an existing audio track,the duplicate being subject to a user selected audio filtration. In thiscase, the original unmodified audio track and a modified track can bothexist. After the user created signal has been created, it can be mixedwith another audio track, e.g., the un-modified track. Alternatively,the user created audio signal is not played back or encoded into thefinal audio signal; rather, its data is merely used to generate timebased video effects.

[0050] When the first component is a picture, it can include stillimages, a film or video. The image can be received by the invention inanalog or digital format and can be live, broadcast, or transmitted. Thepicture signal can be, for example, YUV, YC, RGB, component, compositeor IEEE 1394. The signal could be digital video stored on a local orremote computer. The video can also reside on a pre-recorded medium,such as a DVD, VCD, TIVO, DDR, MiniDV, laserdisc, digibeta, d1, d3 orany medium or apparatus that records images.

[0051] In an alternative embodiment, the system may analyze a video fileto create an output audio file, in which the video file plays normally(sequentially) but the audio file is modified when certaincharacteristics are detected. The system may then play the video in anormal sequential manner and when the video characteristics aredetected, the audio signal is altered. The audio signal modification caninclude: volume, tempo, frequency, sequence, distortion, tone or anyother audio effect. The effect is to view a normal video file while theaudio file is periodically distorted based upon the discrete videosignal.

[0052] One implementation of the invention re-sequences motion pictureframes based upon a secondary media signal. This re-sequencing can beconducted in real time, utilizing live or pre-recorded audio or live orpre-recorded video. Playback can occur in real time, and/or can bestored for later playback.

[0053] The inventive system is particularly useful in conjunction withlive media. In such a case, the media signals processed by the systemmay include live and/or recorded audio and live and/or recorded video.By inputting the live media data, the system can process the media datato incorporate the substitute component or a media effect by utilizingpreviously recorded substitute data when a characteristic is detected.Visual effects that can be implemented during a detection of thecharacteristic can include: varying the camera angle, color, rotation,zoom, pixelation, etc. Alternatively, if multiple cameras are recordingthe same live performance, the signals received from the differentcameras can act as the substitute media signals and data from a second,third, fourth, etc. media source can be inserted when the characteristicis detected. The inventive processing of live media can be configured tooutput live audio with recorded video, live video with recorded audio,or live video with live audio. Of course any other combination wouldalso be within the purview of the present invention.

[0054] Two generalized equations that describe the re-sequencingembodiment when utilizing two media signals are shown below. (1) O(i) =O_(max) F(S₂(i)) Offset calculation (2) S₃(i) = S₁(O(i) + i) Outputmedia Where: S₁(i) = Media signal 1 0 ≦ i ≦ (n₁ − 1) S₂(i) = Mediasignal 2 0 ≦ i ≦ (n₂ − 1) S₃(i) = Output signal 0 ≦ i ≦ (n₁ − 1) F(i) =Filter Function Real Number O(i) = offset at time i 0 < |O(i)| < O_(max)O_(max) = maximum offset (set by user)

[0055] S₁ and S₂ represent independent media signals that are used bythe algorithm to produce an output signal S₃. As discussed, the mediasignals can be virtually any type of media. In the followingdescription, S₁ represents a video signal and S₂ represents an audiosignal. The “i” represents the sequential indicator. The filter functionF can be any linear or non-linear equation that produces a real number.O_(max) is the maximum offset and can be any number that is set by theuser.

[0056] Equation (1) is the offset calculation. The offset calculationsolves the offset based upon the filter function of S₂(i) and has avalue when a user designated characteristic is detected in S₂(i). If acharacteristic is not detected the filter function does not produce anoutput and F(S₂(i))=0. Consequently, O(i)=O_(max) F(S₂(i))=0. If acharacteristic is detected by the filter function, the filter functiongenerates a real number that is multiplied by O_(max) to produce a valuefor O_(i). In a preferred embodiment, the O(i) value is an integer.

[0057] Equation (2) solves the output media S₃(i). If a characteristicis not detected O(i)=0 and the output media signal is the same as thecorresponding media signal 1, S₃(i)=S₁(0+i)=S₁(i). If a characteristicis detected, O(i) has a value, which may be positive or negative. TheO(i) value is added to i to produce an offset value that indicates aframe ahead of or behind the sequential marker “i”. Thus, duringplayback the output media will be the same as media signal 1 until adesignated characteristic is detected in media signal 2. When thecharacteristic is detected, the filter function will produce a value andthe output media will be offset from media signal 1. The output mediamay be output alone or synchronously with media signal 2 or any othermedia signal.

[0058] An example of a common manifestation of the invention is nowdescribed. In this example, the invention is implemented as a set ofsoftware instructions on a computer and two media signals, one audio(S₂) and one video (S₁), are stored on the hard drive of a computer. Thetwo media signals (i.e., the video and the sound) are combined anddisplayed as a new media composition having a sequence of frames, eachcontaining digital data describing, for example, the audio or visualcontent of the frame. When the composition is played, the frames areoutput at a composition frame rate. The composition includes the sourceaudio track (S₂) and an output video track (S₃).

[0059] Referring to FIG. 3., the user, with an input device such as amouse or keyboard, selects the video source (S₁) and the audio source(S₂), at steps S2 and S4, respectively, and places them on a time line.Of course, more than two sources can be selected and processed but forthe sake of simplicity the example only uses two signals.

[0060] The user also selects a characteristic of the audio signal, atstep S6. If no characteristic is selected, default settings for audioprocessing are used. An example of an audio characteristic is afrequency range level, such as 50-100 hz. The audio characteristic canbe sampled in a direct temporal relationship to the frame rate.

[0061] At step S8, the user enters a value for the maximum offset. Thisvariable determines the maximum range from the current frame theinvention will use to select a substitute frame. In other words, thisvalue will determine the maximum offset between the source frame and thecomposition frame to be displayed. Although this variable is describedas a fixed value, this number can also be variable. Specifically, themaximum offset can be a function of any input including a characteristicof the media signal. In other embodiments, the maximum offset can beturned off, disabling the inventive media effects system in the eventthat normal playback is desired.

[0062] At step S10, the filtration function occurs. That is, the audiosource (S₂) is analyzed for the designated characteristic. The filteringmay occur at a user selected sampling frequency. The sampling frequencydefaults to the frame rate of the source picture component (S₁): forNTSC television this is approximately 30 times a second.

[0063] Based upon the filtering function, a data set is determined. Morespecifically, at intervals defined by the sampling frequency, the audiocharacteristic is sampled and normalized. The normalization occurs sothat the substitute frame will not be more than the user selectedmaximum offset.

[0064] In the invention's real time embodiment, step S10 is executed inparallel with steps S16-S18. At step S16, the offset O(i) is calculatedbased upon equation (1). Subsequently, the output signal is calculatedbased upon equation (2). After step S18, it is determined whetherprocessing should continue. If so, the process returns to step S10 andrepeats. If the processing is determined to be completed, the processingends at step S22.

[0065] In an example, a new media composition at 30 frames per second iscreated. The new media composition includes two media signals. The firstmedia source signal (S₁) is a series of 300 images on a hard drive on acomputer. The native playback rate of the motion picture sequence is 30frames per second. The set S₁(i) contains each frame of the series,S₁(1) being the first frame and S₁(300) being the last frame.

[0066] The second media source signal (S₂) is an audio track 10 secondslong. Played from the beginning, the track contains 3 seconds ofsilence, a 60 hz tone with 100% amplitude (level), lasting {fraction(1/30)}^(th) of a second, 3 seconds of silence, a 60 hz tone with 50%amplitude lasting {fraction (1/30)}^(th) of a second, and 3 and{fraction (28/30)}^(th) seconds of silence. S₂(i) contains a set of 300values.

[0067] In this example after the signals are selected at steps S2 andS4, at step S6 the user selects bass tones having a frequency range of50-70 hz as the audio characteristic. The default 30 frames per secondis used for the composition frame rate. At step S8, the user selects themaximum offset to be 60 frames.

[0068] At step S10 the filtering function processes the 300 members ofS₂(i) by sampling the audio characteristic. In this example, the audiocharacteristic is not detected when i=1 through 90. Thus, F(S₂(1))through F(S₂(90))=0. At i=91, F(S₂(91))=100, thus normalizedF(S₂(91))=1. At i=92-182, the audio characteristic is not detected andF(S₂(92)) through F(S₂(182)) are 0. At i=183, F(S₂(i))=60, thusnormalized F(S₂(183))=0.6. At i=184 through 300, F(S₂(i))=0 and thusF(S₂(184)) through F(S₂(300)) are 0.

[0069] Although the values are described as being normalized between 0and 1, it is also possible to use a threshold or non-linear function todetermine the F(S₂(i)) values. In a threshold embodiment, the equationsmay be F(S₂(i))=0 if the detected characteristic has a value<50 andF(S₂(i))=1 if the detected characteristic has a value≧50. Thus, if thedetected characteristic =60 then F(S₂(i))=1. Similarly, any other linearor non-linear formula may be used to define F(S₂(i)) as a function ofthe detected characteristic.

[0070] The output media and offset equations (i.e., equations (1) and(2)) are then solved. As described above, it is preferable to useintegers, thus, the INTEGER function is employed. At i=1 through 90,S₃(i)=S₁ (INTEGER (0*60+i))=S₁(i). At i=91, S₃(91)=S₁(INTEGER(1*60+91))=S₁(151). At i=92 through 182, S₃(i)=S₁(i). At i=183,S₃(183)=S₁(INTEGER (0.6*60+183))=S₁(219).

[0071] The new composition sequence, C(i), is created. It includes 300images played sequentially. C(i) contain the same frames as S₁(i), withthe exception that C(91) would play corresponding frame S₁(151) andC(183) would contain S₁(219).

[0072] Table 1 indicates the values of the variables described in theexample above. TABLE 1 Characteristic i value F(S(i)) O_(max) O(i) C(i) 1-90 0 0 60  1-90 PS(i)  91 100  1 60 151 PS(151)  92-182 0 0 60 92-182 PS(i) 183 60  0.6 60 219 PS(219) 184-300 0 0 60 184-300 PS(i)

[0073] Thus, the sequence of frames in the new media composition is:S(1)-S₁(90), S₁(151), S1(92)-S₁(182), S₁(219), S₁(184)-S₁(300). Notethat S₁(151) and S₁(219) are displayed twice and frames S₁(91) andS₁(183) are not displayed at all.

[0074] Although the invention has been described with reference toseveral exemplary embodiments, it is understood that the words that havebeen used are words of description and illustration, rather than wordsof limitation. Changes may be made within the purview of the appendedclaims, as presently stated and as amended, without departing from thescope and spirit of the invention in its aspects. Although the inventionhas been described with reference to particular means, materials andembodiments, the invention is not intended to be limited to theparticulars disclosed; rather, the invention extends to all functionallyequivalent structures, methods, and uses such as are within the scope ofthe appended claims.

What is claimed is:
 1. A method for resequencing a first input signal,which has a first playback sequence, based upon at least onecharacteristic of a second input signal, the method comprising:analyzing the at least one characteristic of the second input signal;modifying the first playback sequence of the first input signal basedupon the analysis of the at least one characteristic to generate asecond playback sequence; and outputting a third signal comprising thesecond playback sequence.
 2. The method of claim 1, in which the firstinput signal comprises a video signal, the second input signal comprisesan audio signal, and the third signal comprises a video signal.
 3. Themethod of claim 1, in which the first input signal and the second inputsignal comprise the same signal.
 4. The method of claim 1, in which theanalyzing and outputting occur in real time.
 5. The method of claim 1,in which the outputting further comprises outputting the second inputsignal along with the third signal.
 6. The method of claim 1, in whichthe analyzing occurs at a frequency based upon a frame rate of the firstinput signal.
 7. The method of claim 2, in which the first input signalcomprises a plurality of input video signals each having a linearplayback sequence; in which the third signal comprises a plurality ofoutput video signals; in which the at least one characteristic comprisesa plurality of characteristics, each characteristic being associatedwith one of the input video signals; in which the modifying furthercomprises modifying a linear playback sequence of each of the inputvideo signals based upon the analysis of the associated characteristicsto generate an output sequence; and in which the outputting furthercomprises outputting each output video signal in accordance with itsoutput sequence.
 8. The method of claim 2, in which the first inputsignal comprises a plurality of layers; in which the third signalcomprises the plurality of layers; in which the at least onecharacteristic comprises a plurality of characteristics, eachcharacteristic associated with one of the layers; in which the modifyingfurther comprises modifying a layer playback sequence of each of thelayers based upon the analysis of the associated characteristics togenerate an output layer sequence; and in which the outputting furthercomprises outputting each layer in accordance with its output layersequence.
 9. The method of claim 1, in which the modifying furthercomprises altering a frame rate.
 10. A method for resequencing a firstinput signal, which has a series of frames, each having a frame number,based upon at least one characteristic of a second input signal, themethod comprising: analyzing the at least one characteristic of thesecond input signal; for each frame in the series of frames, calculatingan offset based upon the analyzing, the offset being added to the framenumber to obtain an output frame number; and outputting a third signalin a playback sequence using the output frame numbers.
 11. The method ofclaim 10, in which the first input signal comprises a video signal, thesecond input signal comprises an audio signal, and the third signalcomprises a video signal.
 12. The method of claim 10, in which theanalyzing and outputting occur in real time.
 13. The method of claim 10,in which the calculating further comprises calculating based upon a userdefined maximum offset.
 14. The method of claim 13, in which the offsetdynamically varies.
 15. The method of claim 11, in which the first inputsignal comprises a plurality of input video signals each having a linearplayback sequence; in which the third signal comprises a plurality ofoutput video signals; in which the at least one characteristic comprisesa plurality of characteristics, each characteristic being associatedwith one of the input video signals; in which the calculating furthercomprises calculating an offset for each of the output video signalsbased upon the analysis of the associated characteristics to generate anoutput frame number for each output video signal; and in which theoutputting further comprises outputting each output video signal inaccordance with its output frame numbers.
 16. The method of claim 11, inwhich the first input signal comprises a plurality of input layers; inwhich the third signal comprises the plurality of output layers; inwhich the characteristic comprises a plurality of characteristics, eachcharacteristic associated with one of the output layers; in which thecalculating further comprises calculating an offset for each of theoutput layers based upon the analysis of the associated characteristicsto generate an output frame number for each output layer; and in whichthe outputting further comprises outputting each output layer inaccordance with its output layer sequence.
 17. A method for outputtingan output signal based upon at least one characteristic of an inputsignal, the method comprising: analyzing the at least one characteristicof the input signal; controlling an external device based upon theanalysis of the at least one characteristic; and outputting the outputsignal using the controlled external device.
 18. The method of claim 17,in which the output signal comprises a video signal and the input signalcomprises an audio signal.
 19. The method of claim 17, in which theanalyzing and outputting occur in real time.
 20. A computer readablemedium storing a computer program for resequencing a first input signal,which has a first playback sequence, based upon at least onecharacteristic of a second input signal, the medium comprising: ananalyzing source code segment that analyzes the at least onecharacteristic of the second input signal; a modifying source codesegment that modifies the first playback sequence of the first inputsignal based upon the analysis of the at least one characteristic togenerate a second playback sequence; and an output source code segmentthat outputs a third signal comprising the second playback sequence. 21.The medium of claim 20, in which the first input signal comprises avideo signal, the second input signal comprises an audio signal, and thethird signal comprises a video signal.
 22. The medium of claim 20, inwhich the analyzing source code segment analyzes at a frequency basedupon a frame rate of the first input signal.
 23. The medium of claim 20,in which the analyzing source code segment and the output source codesegment process in real time.
 24. The medium of claim 20, in which theoutput source code segment further comprises outputting the second inputsignal along with the third signal.
 25. The medium of claim 21, in whichthe first input signal comprises a plurality of input video signals eachhaving a linear playback sequence; in which the third signal comprises aplurality of output video signals; in which the at least onecharacteristic comprises a plurality of characteristics, eachcharacteristic being associated with one of the input video signals; inwhich the modifying source code segment further comprises modifying alinear playback sequence of each of the input video signals based uponthe analysis of the associated characteristics to generate an outputsequence; and in which the output source code segment further comprisesoutputting each output video signal in accordance with its outputsequence.
 26. The medium of claim 21, in which the first input signalcomprises a plurality of layers; in which the third signal comprises theplurality of layers; in which the at least one characteristic comprisesa plurality of characteristics, each characteristic associated with oneof the layers; in which the modifying source code segment furthercomprises modifying a layer playback sequence of each of the layersbased upon the analysis of the associated characteristics to generate anoutput layer sequence; and in which the output source code segmentfurther comprises outputting each layer in accordance with its outputlayer sequence.
 27. The medium of claim 20, in which the modifyingsource code segment further comprises altering a frame rate.
 28. Acomputer readable medium storing a computer program for resequencing afirst input signal, which has a series of frames, each having a framenumber, based upon at least one characteristic of a second input signal,the medium comprising: an analyzing source code segment that analyzesthe at least one characteristic of the second input signal; acalculating source code segment that, for each frame in the series offrames, calculates an offset based upon the analysis, the offset beingadded to the frame number to obtain an output frame number; and anoutput source code segment that outputs a third signal in a playbacksequence using the output frame numbers.
 29. The medium of claim 28, inwhich the first input signal comprises a video signal, the second inputsignal comprises an audio signal, and the third signal comprises a videosignal.
 30. The medium of claim 28, in which the analyzing source codesegment and the output source code segment process in real time.
 31. Themedium of claim 28, in which the calculating source code segment furthercomprise calculating based upon a user defined maximum offset.
 32. Themedium of claim 31, in which the offset dynamically varies.
 33. Themedium of claim 29, in which the first input signal comprises aplurality of input video signals each having a linear playback sequence;in which the third signal comprises a plurality of output video signals;in which the at least one characteristic comprises a plurality ofcharacteristics, each characteristic being associated with one of theinput video signals; in which the calculating source code segmentfurther comprises calculating an offset for each of the output videosignals based upon the analysis of the associated characteristics togenerate an output frame number for each output video signal; and inwhich the output source code segment further comprises outputting eachoutput video signal in accordance with its output frame numbers.
 34. Themedium of claim 29, in which the first input signal comprises aplurality of input layers; in which the third signal comprises theplurality of output layers; in which the characteristic comprises aplurality of characteristics, each characteristic associated with one ofthe output layers; in which the calculating source code segment furthercomprises calculating an offset for each of the output layers based uponthe analysis of the associated characteristics to generate an outputframe number for each output layer; and in which the output source codesegment further comprises outputting each output layer in accordancewith its output layer sequence.
 35. A computer readable medium storing acomputer program for outputting an output signal based upon at least onecharacteristic of an input signal, the medium comprising: an analyzingsource code segment that analyzes the at least one characteristic of theinput signal; a controlling source code segment that controls anexternal device based upon the analysis of the at least onecharacteristic; and an output source code segment that outputs theoutput signal using the controlled external device.
 36. The medium ofclaim 35, in which the output signal comprises a video signal and theinput signal comprises an audio signal.
 37. The medium of claim 35, inwhich the analyzing source code segment and the output source codesegment occur in real time.